2023-09-28 10:30:33.AIbase.1.8k
The Ma Yi Team Discovers: Fine-tuning Multimodal Large Models Leads to Catastrophic Forgetting
The Ma Yi team proposed the EMT framework to evaluate the catastrophic forgetting of fine-tuned Multimodal Large Models (MLLMs). Experiments showed that while fine-tuning MLLMs improves performance on the fine-tuning dataset, it also leads to a decline in performance on other datasets. During the fine-tuning process, MLLMs generate hallucinated text related to the fine-tuning dataset, ignoring the original problem. This research provides a framework and benchmarks for future work, but the design and training techniques of the model still need further optimization. The Ma Yi team systematically assessed the issue of catastrophic forgetting in MLLMs for the first time, balancing different capabilities.
